Merkel's Christian Democrats received just 26 percent of the vote in elections in North Rhine-Westphalia state, compared with 35 percent in the last vote.
The state-assembly ballot had been dubbed a German mini general election, as one-fifth of all Germans live in the western state.
Merkel, who is personally popular but heads a fractious cabinet, faces her next federal election in 2013.
The opposition Social Democrats and Greens consolidated their hold on North Rhine-Westphalia.
A newcomer, the Pirate Party, which seeks greater Internet access and a state-guaranteed minimum income for all, won just under 8 per cent of the vote, securing seats in a German state assembly for the fourth time straight since September.