Ultimo House was one of Sydney's grand residences, and at the time of its demolition in November 1932 was claimed to be the oldest standing house in Sydney. Built for surgeon John Harris in 1804 using convict labour, the stately two-storey country residence stood on a rise overlooking Blackwattle Creek and Cockle Bay, close to the Parramatta road.
Harris's new house was within the boundaries of the 34-acre (13.8-hectare) grant given to him by Governor King in December 1803. Harris and King had a close relationship: Harris had been appointed by King as his administrative assistant on the recommendation of Lieutenant-Colonel William Paterson and served as a magistrate and Naval Officer for Sydney. Harris's support for King's attempts to control the trade of liquor in the colony was rewarded with three grants between December 1803 and January 1806 totalling 191 acres (77.3 hectares), which comprised much of his Ultimo estate. He increased this through purchase to 233 acres (94.3 hectares) by 1818, taking up most of the peninsula of Ultimo/Pyrmont. 
From the outset, Ultimo House was treated as a country seat by Harris, with the local bush transformed into English-style parkland grazed by imported Indian deer. Although he used the estate for some small-scale mixed farming, the Ultimo Estate was largely for show rather than production.
After King departed, Harris lost favour with his successor, Governor Bligh. In January 1808 Harris held a dinner at Ultimo House with fellow officers opposed to the governor. It was from this dinner that the conspirators returned to Sydney to call the troops out to arrest Bligh in what was to become known as the Rum Rebellion. 
In 1814 Harris, by now out of the navy, returned to Sydney from London where he had been giving evidence at the court-martial of Captain Johnston for his role in the overthrow of Bligh. Harris returned to Ultimo House with his new wife and soon afterwards commissioned Francis Greenway (who had arrived as a convict in 1814 on the General Hewitt, the same ship Harris had returned on) to extend the property. This was Greenway's first private commission. 
The colony's premier address
Harris moved out of the house in 1821, leasing the property to a number of prominent tenants who maintained it as one of the colony's premier addresses. Edward Riley leased the house and estate between 1821 and 1824, after which Judge John Stephen took up residence; he in turn was followed by John Edye Manning, registrar of the Supreme Court of New South Wales.
During Riley's occupancy, the house became one of Sydney's foremost social spots with a number of large parties reported in the press. At one to send off HMS Tees and her crew, 240 people dined in three shifts while Captain Piper's band played dances until the morning. 
Although the house was primarily used as a domestic residence, for a short period in the mid 1850s the property was employed by the Australian Washing Association as the site for their Sydney laundry. The Australian Washing Association advertised that their washing house was fitted with hot and cold water, with articles for washing being taken in Monday to Wednesday and delivered back Thursday to Saturday. 
By 1855 the house was once again a residence, leased to wool broker James Wallach, and by 1858 it was occupied once more by members of the Harris family, first by Mary Ann Harris and then George Harris, Surgeon Harris's nephew, who lived there until his death in 1897. During his occupancy much of the estate was subdivided, leaving the house on approximately three acres (1.2 hectares) by the late 1890s. George Harris also modified the house, adding three turreted towers to the façade, probably after a fire in the 1870s. 
Pressure for the house to be resumed for public use first arose in 1883, when the newly appointed Board of Technical Education held its first public meeting. The Board was to administer the Sydney Technical College, established in 1878 by the Sydney Mechanics' School of Arts. At the meeting it was suggested that the Board purchase 3.5 acres (1.4 hectares) of Harris's estate to build a new Technical College. Although initially rejected by the Minister for Public Instruction, Joseph Carruthers, the site was purchased in 1891 and a grand new technical college, technical museum and two high schools were constructed, fronting Harris and Mary Ann streets.
In 1910 the technical college doubled its size, expanding to take in the entire block of Harris, Mary Ann, Jones and Thomas streets. This included Ultimo House and two acres (0.8 hectares) of gardens associated with it. The first use of this site was as a lunch and recreation area for the students, as the gardens provided some of the only open space and tree coverage in the neighbourhood. The house itself was refashioned for use as classrooms for women's dressmaking, millinery and flower-making, veterinary science, and agriculture, and as general lecture rooms. The stables and coach house at the rear of the house, although dilapidated, were also transformed into classrooms for bricklaying, plastering, masonry, drawing and design.  The transformation of the old house and its outbuildings into classrooms was supervised by the head of the architecture department, James Nangle. Curiously, Ultimo House had already been associated with further education in Sydney, when in 1900 Mrs George Harris of Ultimo House had established a scholarship at the nearby University of Sydney for the study of law, providing £1700, with an annual scholarship of £50. Founded in honour of her late husband, the George and Matilda Harris Scholarship continues to support undergraduate law students at the University of Sydney.
In November 1932, Ultimo House was demolished to make way for further expansion at the Technical College, in the form of a new electrical engineering building. At the time of demolition, a number of articles appeared in the Sydney Morning Herald lamenting the end of a Sydney landmark. 
Shirley Fitzgerald and Hilary Golder, Pyrmont & Ultimo: under siege, 2nd edition, Halstead Press, Sydney, 2009
J Broadbent and J Hughes, Francis Greenway: Architect, Historic Houses Trust, Sydney, 1997