A public service insider has trashed the Prime Minister’s attempts to tackle indigenous affairs.

Reports say Tony Abbott’s stated fondness for the Aboriginal community has not been able to shine through the cloud of hundreds of specialist public servants sacked and funding cuts to key programs.

With vital efforts stalled and staff morale in the “doldrums”, disgruntled public servants say they want to work to improve indigenous affairs, but are being hindered by overhaul and upheaval.

There is speculation that the Prime Minister may soon cut short his week-long trip to Arnhem Land, part of his pledge to be “a Prime Minister for indigenous affairs”.

The trip may seem less meaningful though, knowing that the departments that would implement any promises he makes are in a state of disarray.

As part of Mr Abbott’s takeover of all Indigenous Affair functions, senior leaders in the Prime Minister and Cabinet (PM&C) department's Indigenous Affairs Group moved over to Canberra's dress circle, which is about 10 kilometres away from the workers they oversee.

Not only this, but staff say they are struggling to deal with repeated restructures since the beginning of this Government’s term. They say it is difficult to tackle a growing task when the new Government forces in an extra 1800 public servants in a merger.

Speaking to Fairfax Media, staff said they were being made to do much more with much less.

Workers in the PM&C's Indigenous Affairs Group are trying to overhaul programs and move key responsibilities to regional offices to fulfil the PM’s promised shake-up of services, but it is made difficult by several restructures and 236 redundancies, which have seen senior executives, middle-managers wasting time by re-applying for their own jobs.

“The disorganised state has meant that indigenous groups and other stakeholders have not been consulted,” one staffer allegedly told Fairfax.

“Most 2014-2015 money is not available as many existing contracts were automatically extended for 12 months.

“There are some funds for indigenous education projects but this will be well short of demand.

“Staff morale is in the doldrums,” they said.

The Indigenous Affairs workers say Associate Secretary Liza Carroll and Deputy Secretary Richard Eccles are now based at PM&C's Barton HQ in Canberra's stylish inner-south, while the bulk continues to work 9.5km away in the less thrilling suburb of Woden.

A PM&C spokesperson told reporters that the department was doing its best after being forced to quadruple in size, absorbing 1800 public servants from nine other agencies as part of the Prime Minister’s takeover as Minister for Women as well.

He said any service providers with contracts ending up to 30 June 2014 and “who had an expectation of ongoing funding” will have their contracts extended.

The spokesperson added that “the government will work in partnership with Indigenous communities to tailor action in the priority areas of schooling, jobs and community safety”.