You cannot leave "temperature outside", as temperature is the key factor to know if the balloon would pop or not.
Let's set up some assumptions about the problem so we can calculate something:
The gas inside the balloon behave according the ideal gas law: $PV=nRT$
That the balloon can hold a differential pressure of 30 mmHg (=4000 Pa) as found in this experiment.
- That the maximum volume of a balloon is 0.015 m$^3$ as suggested here.
- The balloon starts filled to its maximum size.
With this we can calculate how many moles of gas the balloon can hold when inflated in space at a given temperature.
For the temperature I'll assume it will be something between the temperature of the cosmic microwave background radiation (if it is on the shade) of ~2.7 K (-270 °C), and the temperature that a black body that is illuminated by the Sun near Earth's orbit, which would be about 278 K (about 5°C). This assumes that the balloon is small enough so that heat can be transferred efficiently by diffusion from one side to the other (the calculation of this value can be seen in the code at the end and is based on the equation here).
With this range of temperatures we can find the volume of the balloon on the surface assuming a final pressure of 1 atm (101325 Pa) and a temperature of 15 °C.
We get the following relationship:

In red are the final volumes for which the balloon would explode and in blue when it wouldn't. The critical temperature in between is -262 °C (~11 K).
Therefore, the balloon would not pop as long as its initial temperature is higher than -262 °C.
This also assume that the hottest point in the trajectory is the surface. This means that we are assuming that the balloon will pass through the thermosphere fast enough to do not reach thermal equilibrium with it (it can be extremely hot). This could be a reasonable assumption given the very low density of that layer.
Of course this also assume the rubber of the balloon do not change its properties with temperature.
So, only once you specify the height at which you release the balloon and its initial temperature, then you can start calculating fall time. And that will be more difficult, because even if you assume the balloon is at free fall terminal velocity the whole time, this velocity will change along the fall line due to changes air density and balloon size. Air density and balloon size will affect air resistance and, therefore, terminal free fall speed. You will need to do an analytical or numerical integration of the fall to get the total time.
Matlab code used to generate the above figure
R=8.3145;%Ideal gas constant [J/mol K]
Lsol=3.828e26;%Solar luminosity [W]
Dsol=149597870700;%Distance earth-sun [m]
sigma=5.670373e-8;%Stefan–Boltzmann constant [W m^-2 K^-4]
alpha=0;%albedo
epsilon=1;%emissivity
% Temperature of a black body (rotating or small enough to transfer heat efficiently from the illuminated side to the dark side)
Tbb=((1/4)*((Lsol*(1-alpha))/(4*pi*epsilon*sigma*(Dsol^2))))^(1/4);
% Temperature range
T=linspace(2.7,Tbb,1000);%[K]
Vmax=0.015;% Exploding volume [m³]
Pspace=4000;%Maximum pressure hold by the balloon [Pa]
n=Pspace*Vmax./(R*T);% number of moles that the balloon would hold
Patm=101325;% Atmospheric pressure [Pa]
Tsurface=15+273.15; % temperature at the surface [K]
% Volume once back on earth surface
Vfin=n*R*Tsurface./Patm;
explodeIdx=Vfin>Vmax;
Tlim=find(explodeIdx,1,'last');
figure('Color','w');
hold on
box on
plot(Vfin(~explodeIdx)*1000,T(~explodeIdx)-273.15,'b','LineWidth',2);
plot(Vfin(explodeIdx)*1000,T(explodeIdx)-273.15,'r','LineWidth',2);
plot([0 max(Vfin)]*1000,[1 1]*T(Tlim)-273.15,':r','LineWidth',2)
ylim([0 max(T)]-273.15);
xlabel('Final volume [liters]')
ylabel('Initial temperature [°C]')