Things had looked promising for Hege Riise’s side, after a second half brace from Ellen White had cancelled out Alanna Kennedy’s opener.
But Chelsea’s Sam Kerr restored parity late on, before an enthralling first period of extra time saw a Caroline Weir penalty saved by Teagan Micah, minutes before 18-year-old Mary Fowler’s deflected drive put the Australians ahead.
Kerr added a fourth on 105 minutes but White completed her hat-trick to give Team GB hope.
However, it was to no avail, with the Matildas facing Sweden in Yokohama for a place on the medal podium.
Eight of our City stars were selected in Hege Riise’s starting XI, with Ellie Roebuck’s goal protected by Steph Houghton, Lucy Bronze and Demi Stokes.
Keira Walsh and Weir started in midfield, while Lauren Hemp, Rachel Daly and Kim Little provided support for White at the sharp end of the pitch.
And the chances were flowing early on, with Walsh twice going close inside the first 15 minutes, including smacking a long-range effort against the post after being afforded space on the edge of the Matildas’ box.
It was Hemp who was next to trouble Micah’s net as Hege Riise’s side maintained their dominance, stinging the palms of the Australia stopper after reacting quickest to Daly’s looping cross, which had come back off the far post.
However, it was the Australians who struck first just after the half hour with their first effort on the Team GB goal when Kennedy was on hand to power home a header from Steph Catley’s corner.
The goal gave the Australians a new lease of life, and they enjoyed the majority of the early exchanges following the restart in Kashima.
But Hege Riise’s side were far from on the back foot, with Hemp once again looking a real threat, particularly when cutting in from the flank.
And the winger turned provider for White before the hour to hand Great Britain a crucial equaliser, picking out her teammates from the right.
With her first real chance of the game, the City forward stole a yard between the Australian defenders and glanced an inch-perfect header into the far corner of the net for her fourth goal of this Olympic games.
It was just rewards for Great Britain, who brought Stokes off before the hour after an industrious display at left-back following the goal, but they rode their luck soon after.
Minutes after the equaliser, a deep cross from the right landed at the feet of Australia’s Caitlin Foord, whose ball back across goal was inadvertently deflected onto the bar bu Houghton.
The City skipper’s blushes were spared and she and her teammates were sent into raptures soon after when White struck again with 25 to play.
A long throw caused havoc in the Australia box with pressure from Hemp on Kennedy and Aivi Luik eventually seeing the ball pop out into the path of the City forward.
It was the kind of chance that a player of her calibre would dream for, and she made no mistake with a powerful drive into the bottom corner of Micah’s net, despite the stopper getting a hand to the effort.
After getting their noses in front, Great Britain looked to preserve their lead by bringing on Jill Scott with ten minutes to play, as Australia gradually began to grow back into the game.
But the pressure eventually told in the 89th minute when Kerr plucked an aerial ball out of the sky before spinning and firing a vicious low drive past Roebuck to force extra time.
The ongoing battle between Walsh and Micah continued, with the stopper tipping the midfielder’s rasping drive over the bar soon after the restart, before she pulled off a spectacular save soon after to prevent White from grabbing a third from close-range.
Hege Riise rolled the dice soon after, bringing on former City forward Nikita Parris for Hemp, while Walsh was also replaced by Sophie Ingle.
And Parris was in the thick of the action almost immediately, winning a penalty after being hauled down by former Olympique Lyonnais teammate Ellie Carpenter.
Weir took on the responsibility from the spot but Micah’s standout performance continued as the goalkeeper guessed right to palm away the City midfielder’s effort.
And the game was turned on its head completely when substitute Fowler brought the ball under her spell and fired towards goal, with her effort taking a wicked deflection off Bronze to nestle into the far corner of the net.
As the legs began to tire, Kerr earned the Matildas a two-goal buffer seconds after the restart, giving Roebuck no chance with a clinical header.
The game looked to be getting away from Great Britain, and Georgia Stanway was brought on for Bronze with ten to play to try and regain some impetus.
And hope was restored when White completed her hat-trick with another glancing header, lobbing Micah from Fran Kirby’s searching near-post cross.
Great Britain continued to knock on the Australian door but, despite their best efforts, their dreams of a medal are over at the quarter-final stage.
Australia will face Sweden in Yokohama on Monday 2 August following their 3-1 victory over host nation Japan.
