Objects can change in size when their temperature changes. I'm wondering whether the cooling of the earth's core will affect it's size? Or will the change be insignificant due to the elements and states involved or the pressure acting on the core etc.?
The cooling of the Earth should affect it's size, everything else being equal. How much, gets complicated, and there are other factors. Perhaps the cooling isn't even the biggest one.
Looking at a density diagram, (source, Wikipedia)
There is a jump in density between the liquid and solid core, which makes sense. For the most part, an element in solid state is dense than in liquid state (Ice/Water is a rare exception). By this table below, the bottom of the liquid core (density 12.2) is less dense than the top of the solid core (12.8). If we use that as a rule of thumb, that's a 5% increase in density as the phase of the Iron-Nickle-otherstuff transitions from solid to liquid.
Running the Math, the radius of the inner core, 1,220 km and outer core, 3,400 km. Volume = 4/3 πr³, increase the density of the inner core as it solidifies by 5%, you end up with a bit under 3,350 km, or a reduction in volume of about 7.5 billion cubic KM (just by solidifying, not doing temperature yet). If you remove that much volume from the entire Earth though, it works out to 15 KM.
My guess is that under such high pressure, the temperature contraction would be less, but that's just a guess. I found a few articles on the subject, pay articles mostly but nothing that neatly estimated temperature to density at 1 million atmospheres or more. That's the crux of your question, but I couldn't find anything specifically on temperature expansion at those pressures. I did find (and this is fairly obvious looking at a density of the Earth chart), that pressure determines density far more than temperature, inside the Earth. We think of solids as mostly incompressible, but that's no longer true with millions of ATMs of pressure. Solids do measurably compress at those pressures. This article says that pressure, not temperature is the primary driver for the core's density.
Jupiter, as a fun example, shrinks about 2 CM per year. But much of it's size is gas, not solid, which is much more responsive to temperature changes.
There's other factors too. Earth's rotation increases it's volume "slightly".
Radioactive decay both helps maintain the Earth's interior temperature and it also releases gas into the core, mantle and lithosphere as a byproduct of that radiation, primarily Helium and Argon. (but if I was to guess, I think that's probably a minor factor in overall density and size and I think, ignorable). Lead is less dense than Uranium, but Calcium is more dense than Potassium, so the decay works both ways for density in that sense.
Another factor for overall size is the changes to the lithosphere. Carbon capture and the formation of Granite over time, as well as the formation of soil, all tends to make the earth's lithosphere more puffy. But, taking that into account, the earth's Crust is about 30-50 km thick, over land only (29% of the earth). If we figure the Earth's Crust is about 40% expanded, over the entire earth, that works out to about 4-6 KM crustal expansion due to the oxidation of basalt rock and the formation of continental crust, probably a fair bit less than the contraction over the same period of time.
One rather simple way to measure changes in size of the Earth is to measure it's change in rotation, which slows over time due to the moon, but can also change as a result of structural changes such as Earthquakes, or melting ice and growing oceans. While this may seem to have nothing to do with your question, Earthquakes are, in a sense, settling of the Earth, and large Earthquakes like the 2011 Japan Earthquake did speed up the Earth's rotation due to the Earth becoming slightly more compact.
I think it's likely that the Earth grows more compact over time, but as Gordon Stranger said, it's very slow, and the Sun will go red giant long before the Earth's core is cooled.
How much? If we say 15 KM due to solidifying the inner core, maybe another 10-15 KM for solidifying of much of the mantle, add (Gordon Stranger's estimate) of 10 KM due to temperature, and maybe add 1-2 KM due to the Earth's rotation slowing . . . that's a hugely ballpark guess, but adds up to about 21-32, but a safer, rounder estimate, maybe 20-40 KM as a bad guess accounting for some settling, but that's if the Earth is allowed to cool on it's own over a long enough period of time, like, I don't know, 50-100 billion years. That works out to roughly 1/2 of 1%. Works out to about 1/2 of 1% of it's radius. Earth's gravity would increase by about 1% as a result, 1.5% near the equator.
Hope that wasn't too much guessing. This might be a tough one to get a definitive answer on, but I'm pretty sure you're right that contraction is the overall direction.
Over the 4.6 billion years of the Earth's evolution, so far, the inner/outer core interface has "cooled" to a mere 6000 deg C. Of course, other things being equal, the core will contract as it cools, but the rate of cooling will be massively delayed by the slow radioactive decay of uranium 235, U 238 and Thorium. Delayed, in fact by many more billion years. The mantle will cool with it, and being plastic, will deform to take up the space of the thermal contraction. However, by the time the radioactive heating decays to much lesser values, the sun will have blown up into the red giant phase. Even if the Earth is not actually engulfed by the expanded sun, it will be so heated that the oceans, and more volatile fractions of the crust will have boiled off into space, to become part of the enhanced solar wind.
If we assume that the Earth's core will eventually cool by 4000 deg C, then Earth will eventually have a reduction in radius of about 10 km (my very rough back of an envelope calculation) from thermal contraction alone. How much of the surface will have boiled off into space by then is anybody's guess.