916098 - ZTE open fails to apply system update

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Description

[Kai Engert (:KaiE:)]

> I have one of the ZTE Open phones sold by ebay UK.
> 
> Today an OTA system update was offered to me, 9 MB.
> I approved installation, system rebooted and started install, progress
> indicator was moving until 40%, then I stopped looking at the phone for
> a moment.
> 
> Next I saw that phone had rebooted into recovery.
> It says:
>  Verifying current system...
>  assert failed: apply_patch_space (17879708).
>  E:Error in /sdcard/updates/fota/update.zip. 
>  Installation aborted.
> 
> Phone as a 16 GB card, which was freshly formatted prior to inserting it
> into the ZTE open, and which has just a few music files on it, so it's
> mostly empty. The card had worked fine in other phones previously.


The phone did successfully reboot after showing above error.

I'll attach the recovery.log found on the sd card.

Comment 1

[Kai Engert (:KaiE:)]

Attachment: recovery.log 1 (Kai's failed log)

Comment 2

[Kai Engert (:KaiE:)]

$ sha256sum 0/update.mar fota/update.zip 
e49e75cd47c2d7dbe34ae8fbf3a1850b4d2bb083c5103e845d6ae6c2409fe026  0/update.mar
a073fdaff8b91ec3604b75cf90d0e32fcc3c3324aca3d0922a4a5cdc586f5308  fota/update.zip

$ sha1sum 0/update.mar fota/update.zip 
795b9eb0ebe0a6f4cb23a64ee904fe7d0ebe05b1  0/update.mar
960295330fcd1cc0007fdc39d163af559f6c33f0  fota/update.zip

Comment 3

[Kai Engert (:KaiE:)]

I retried the update, and apparently it worked now.

So you should figure out why mounting the cache failed the first time.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Ivan, this appears to be a new issue with the ZTE FOTA update released today.  Just wanted to make sure you saw it.  Thanks!

Comment 5

[Ivan Tsay (:ITsay)]

Hi Ben,

I've reported the issue to ZTE for them to check this issue.

Comment 6

[Bill Gianopoulos [:WG9s]]

I have the same issue.

Comment 7

[Michelle Luna]

I have two users on SUMO with same issue and same error over the weekend: https://support.mozilla.org/en-US/questions/971252#answer-479515

Comment 8

[Felipe D.]

I am having the same issue as described above and on the SUMO thread.  I am using the ZTE Open sold on their US ebay store, with the included sdcard they threw in (I was among first 100 buyers).

Since there were reports that a second time could do the trick, I've tried again to make sure. I think I've tried at least 3 times now, always with the same result. On the recovery screen I simple reboot the phone rather than choosing any other option.

The first time I did get a message telling me the update failed. Subsequent attempts did not notify my of a failure. Each time, a minute or so after the phone reboots, I get prompted to download & install updates: 1 system update, 2 apps.

Thanks.

Comment 9

[Felipe D.]

Small correction, I do seem to get the error[1] after rebooting. Maybe I just hadn't noticed the other times.


[1] "There was an error while downloading the updates."

Comment 10

[Bill Gianopoulos [:WG9s]]

(In reply to felideon from comment #8)
> Since there were reports that a second time could do the trick, I've tried

I think there was confusion here.  On the second attempt to boot, the phone will boot, but without applying the update.

Comment 11

[Kai Engert (:KaiE:)]

> I think there was confusion here.  On the second attempt to boot, the phone
> will boot, but without applying the update.

As stated in comment 3, it worked for me on the second attempt.
The sequence of events was:

- I tried to update
- update failed, phone showed the recovery screen
- I selected "reboot"
- the phone booted normally, but gave the short popup notice "update failed"
- I tried to update again
- this time the update operation completed successfully

Comment 12

[Michelle Luna]

I should add that I was able to update one of my own US Ebay ZTE Open devices with the system update. I'm in SF today if someone wants to take a look at it to see what is different. (I have one device that succeeded and one that fails).

Comment 13

[Kai Engert (:KaiE:)]

My guess is there's a race condition during boot, which causes mounting of the cache partition to fail occassionally.

Comment 14

[Kai Engert (:KaiE:)]

I propose you should find out if everyone fails because of the same problem.

You could insert the SD card into a computer's card reader, inspect the file recovery.log, and check if you see the same failure as in the attachment.

Failed to mount /dev/block/mtdblock4 on /cache: Device or resource busy
mtd mount of cache failed: Device or resource busy
...
script aborted: assert failed: apply_patch_space(17879708)
assert failed: apply_patch_space(17879708)

If you don't have the above failure, it might be helpful to attach your recovery.log file to this bug.

Comment 15

[Michelle Luna]

I don't have sdcards installed in either device. I will try to take a look at my recovery.log after lunch and post back with what I find.

Comment 16

[Michelle Luna]

Attachment: recovery.log 2 (Michelle's failed log)

recovery.log with apply_patch_check failure

Comment 17

[Bill Gianopoulos [:WG9s]]

So, do we have any idea what is going on here?  We have a report in a different bug, I will try to find ad reference, that with one of the second round ZTE phones on ebay in the US without being, used yet with no sim and an empty 32gig sdcard installed the update works just fine.

So do we need an empty sdcard?  the newer phone or restore to factory default to do the update?

Comment 18

[Bill Gianopoulos [:WG9s]]

Was bug 890974 comment 61.

Comment 19

[Ivan Tsay (:ITsay)]

I got the initial analysis from ZTE about the issue.
1. According to log1, update fails because build.prop file in the phone is modified so that when the failure is on the file verification.
2. According to the log2, as it is described in the bugs that the space of cache partition is not enough so that the uncompressed update files cannot be store in the cache partition for the update.

For the log1 issue, ZTE need to know if the files of the phone is somehow modified to cause the error.
For the log2 issue, user needs to enter recovery mode ti wipe the data in cache partition and try the update again.

Comment 20

[Kai Engert (:KaiE:)]

Attachment: recovery log (Kai's successful log), same phone as in recovery log 1, (attachment 804433), worked after a simple reboot, no wiping, unclear why more memory available was suddenly available

(In reply to Ivan Tsay (:ITsay) from comment #19)
> I got the initial analysis from ZTE about the issue.
> 2. According to the log2, as it is described in the bugs that the space of
> cache partition is not enough so that the uncompressed update files cannot
> be store in the cache partition for the update.
> 
> For the log2 issue, user needs to enter recovery mode ti wipe the data in
> cache partition and try the update again.


Comment 19 mixes the numbers of log1 and log2 with those of the attachments, which might make this complicated to read...

The analysis for my logfile ignors the fact that it worked on second attempt.
I *didn't* wipe the cache, and it worked anyway on second attempt!

All I did was reboot and try again!
I will attach the logfile for the succeeding update.

In both attempts, mounting of the cache partition failed.

Why was the available memory different in the second attempt?
13.5 MB in first attempt.
55.7 MB in second attempt.

That's probably what you need to find out.

Comment 21

[Ivan Tsay (:ITsay)]

(In reply to Kai Engert (:kaie) from comment #20)
> Created attachment 807674 [details]
> recovery log (Kai's successful log), same phone as in recovery log 1, 
> (attachment 804433 [details]), worked after a simple reboot, no wiping,
> unclear why more memory available was suddenly available
> 
> (In reply to Ivan Tsay (:ITsay) from comment #19)
> > I got the initial analysis from ZTE about the issue.
> > 2. According to the log2, as it is described in the bugs that the space of
> > cache partition is not enough so that the uncompressed update files cannot
> > be store in the cache partition for the update.
> > 
> > For the log2 issue, user needs to enter recovery mode ti wipe the data in
> > cache partition and try the update again.
> 
> 
> Comment 19 mixes the numbers of log1 and log2 with those of the attachments,
> which might make this complicated to read...
> 
> The analysis for my logfile ignors the fact that it worked on second attempt.
> I *didn't* wipe the cache, and it worked anyway on second attempt!
> 
> All I did was reboot and try again!
> I will attach the logfile for the succeeding update.
> 
> In both attempts, mounting of the cache partition failed.
> 
> Why was the available memory different in the second attempt?
> 13.5 MB in first attempt.
> 55.7 MB in second attempt.
> 
> That's probably what you need to find out.

TE phone has around 57MB for the cache partition as far as I know. The question is why the cache partition was almost used up and only 13.5MB left over in the first attempt. I can forward this message to ZTE for further check on this in which circumstance the cache will be used besides FOTA.

Comment 22

[Felipe D.]

Attachment: 'manually' applying the update from recovery mode screen after wiping cache partition

(In reply to Ivan Tsay (:ITsay) from comment #19)
> For the log2 issue, user needs to enter recovery mode ti wipe the data in
> cache partition and try the update again.

I have a log2 type error when trying to apply the update. So when I got to the recovery mode screen, I performed a "wipe cache partition" followed by a "reboot system now". Then I tried the download/install again and same error.

So the second time I got into recovery mode, I performed a "wipe cache partition" immediately followed by a "apply update from external storage", selecting the /sdcard/updates/fota/update.zip file.  At that point I get a different error, attached.

Could you clarify how to "try the update again" or were you expecting a normal rinse/repeat after the wiping to work?

Thanks.

Comment 23

[Bill Gianopoulos [:WG9s]]

Is this something that someone will be at the summit in Santa Clara who can figure out why this does not work for me?

Comment 24

[Bill Gianopoulos [:WG9s]]

Telling it to wipe cache then rebooting and retrying the update still fails.

Comment 25

[Ivan Tsay (:ITsay)]

(In reply to Bill Gianopoulos [:WG9s] from comment #24)
> Telling it to wipe cache then rebooting and retrying the update still fails.

Is it possible to retrieve build.prop for our partner to further check the verification issue? Regarding the analysis of new log, our partner said there is no new finding from it.  I've request our partner to directly join the discussion in this bug for efficient communication.

Comment 26

[Bill Gianopoulos [:WG9s]]

(In reply to Ivan Tsay (:ITsay) from comment #25)
> (In reply to Bill Gianopoulos [:WG9s] from comment #24)
> > Telling it to wipe cache then rebooting and retrying the update still fails.
> 
> Is it possible to retrieve build.prop for our partner to further check the
> verification issue? Regarding the analysis of new log, our partner said
> there is no new finding from it.  I've request our partner to directly join
> the discussion in this bug for efficient communication.

Sure!  just give me steps to do that.

Comment 27

[Bill Gianopoulos [:WG9s]]

i will also be in Santa Clara at the Summit with this phone if someone will be there with more knowledge and better debugging tools.

Comment 28

[xiongping]

Hi Bill, 

Please do following steps to retrieve build.prop for our analysis :

Step 1: Install ADB tool on the computer
Download the ADT bundle, please refer to http://developer.android.com/sdk/index.html
Unpack the zip file, and the ADB tool will be found in the directory of (for example) E:\Software\adt-bundle-windows-x86-20130729\sdk\platform-tools

Step 2: Set the phone to ADB mode
Setting->Device Information->More Information->(Developer Setting)Developer->（Select）Remote debugging

Step 3：Pull the build.prop file to the computer
//If the adb setting is OK, the roamer2 device will be showed as below

E:\Software\adt-bundle-windows-x86-20130729\sdk\platform-tools>adb devices
adb server is out of date.  killing...
* daemon started successfully *
List of devices attached
roamer2 device


E:\Software\adt-bundle-windows-x86-20130729\sdk\platform-tools>adb pull /system/build.prop e:\build.prop
179 KB/s (5750 bytes in 0.031s) 

After having build.prop , we will compare it with original one and check whether it was changed before FOTA update .

Thanks.

Comment 29

[xiongping]

(In reply to Felipe D. from comment #22)
> Created attachment 807737 [details]
'manually' applying the update from
> recovery mode screen after wiping cache partition

(In reply to Ivan Tsay
> (:ITsay) from comment #19)
> For the log2 issue, user needs to enter
> recovery mode ti wipe the data in
> cache partition and try the update
> again.

I have a log2 type error when trying to apply the update. So when I
> got to the recovery mode screen, I performed a "wipe cache partition"
> followed by a "reboot system now". Then I tried the download/install again
> and same error.

So the second time I got into recovery mode, I performed a
> "wipe cache partition" immediately followed by a "apply update from external
> storage", selecting the /sdcard/updates/fota/update.zip file.  At that point
> I get a different error, attached.

Could you clarify how to "try the update
> again" or were you expecting a normal rinse/repeat after the wiping to work?
> Thanks.

Please perform a "wipe cache partition" immediately followed by a "apply update from external storage" . It's not necessary to reboot the device before applying update .

Comment 30

[Dave Hylands [:dhylands]]

I believe that I may have identified at least part of the problem. Please see bug 858188.

For the time being, I think what this means is that you need to remove the sdcard in order to apply the update.

Bill - I'll be in Santa Clara for the summit as well.

Comment 31

[Felipe D.]

(In reply to Dave Hylands [:dhylands] from comment #30)
> For the time being, I think what this means is that you need to remove the
> sdcard in order to apply the update.

That doesn't work. I tried this last week as well. I can try again and give you more info (recovery.log) if you'd like.

Comment 32

[Felipe D.]

(In reply to xiongping from comment #29)
> Please perform a "wipe cache partition" immediately followed by a "apply
> update from external storage" . It's not necessary to reboot the device
> before applying update .

Thank you for clarifying. As I mentioned, I did try that and got the error here:
https://bug916098.bugzilla.mozilla.org/attachment.cgi?id=807737

I will attach the build.prop in a bit.

Comment 33

[Felipe D.]

Attachment: build.prop -- ZTE Open (first round US eBay store)

The build.prop file as promised.

Comment 34

[xiongping]

Hi Felipe , 

Thanks for providing build.prop . I compared it with original one and found that there is a difference . The ro.build.display.id should be OPEN_US_DEV_FFOS_V1.0.0B01 , but we find it has been changed to OPEN_US_DEV_FFOS_V1.0.0B02 in your build.prop . Because the content of build.prop has changed , it will make the sha1 hash of build.prop different from sha1 hash of original one which will cause the FOTA update abort . 

Did you root the device before FOTA update? Normally , the ro.build.display.id property is read only , you can not change it .

Thanks

Comment 35

[jezra]

I have the exact same issue and my device is from the first US ebay sale. The OS on the device is OPEN_US_DEV_FFOS_V1.0.0B02 and that is how the device was shipped.

Comment 36

[xiongping]

(In reply to jezra from comment #35)
> I have the exact same issue and my device is from the first US ebay sale.
> The OS on the device is OPEN_US_DEV_FFOS_V1.0.0B02 and that is how the
> device was shipped.

Hi jezra , 

We will investigate this issue internally and let you know if there is any progress . 

Thanks.

Comment 37

[Felipe D.]

(In reply to xiongping from comment #34)
> Did you root the device before FOTA update? Normally , the
> ro.build.display.id property is read only , you can not change it .

Interesting. But no, sorry, I did not root the phone or anything.

Thanks for looking into this.

Comment 38

[Bill Gianopoulos [:WG9s]]

My build.prop file is identical to the one attached already.

I wonder if this is related to the issue with the bogus update ZTE had posted previously.  See Bug 908512.  I wonder if the attempt to install that ended up altering the ro.build.display.id.

Comment 39

[xiongping]

(In reply to Bill Gianopoulos [:WG9s] from comment #38)
> My build.prop file is identical to the one attached already.

I wonder if
> this is related to the issue with the bogus update ZTE had posted
> previously.  See Bug 908512.  I wonder if the attempt to install that ended
> up altering the ro.build.display.id.

yes , the bogus update package will alter the ro.build.display.id . I double check it with my colleague. They told me that the package was uploaded to the server by mistake .

Comment 40

[Bill Gianopoulos [:WG9s]]

(In reply to xiongping from comment #39)
> yes , the bogus update package will alter the ro.build.display.id . I double
> check it with my colleague. They told me that the package was uploaded to
> the server by mistake .

OK,so now that we know what happened, hopefully a fix or workaround will be forthcoming soon?

Comment 41

[Felipe D.]

(In reply to Bill Gianopoulos [:WG9s] from comment #40)
> OK,so now that we know what happened, hopefully a fix or workaround will be
> forthcoming soon?

Presumably, if you root the phone it should be possible edit the build.prop file? But I'd rather not have to do that, as I want to experience this round of OTA to see what it enables (fastboot?) and/or changes. At that point I wouldn't mind going with experimental builds.

Comment 42

[Felipe D.]

(In reply to xiongping from comment #36)
> We will investigate this issue internally and let you know if there is any
> progress.

Any news on this?

Comment 43

[jezra]

Support seems to be diluted and spread out over various websites. 
There is an 'answer' on one of the support pages that may help you.

https://support.mozilla.org/en-US/questions/971252?page=2#answer-484863

Comment 44

[Felipe D.]

Right, that would help me do all this manual. I'm just wondering if the OTA will ever be fixed. Not sure if it's worth "pretending I'm a lay user" at this point since clearly the ZTE Open was not ready for consumer use.

Hopefully FirefoxOS brand isn't tarnished in Colombia/Venezuela/Spain.

Comment 45

[jezra]

Since a 'fix' already exists, ZTE will probably not put the resources into recreating the fix as an OTA update. 

Basically, we paid money to be beta testers of a product with no support and no upgrade path other than hoping the OEM sends us some code that may or may not work properly. Hopefully in the future Mozilla can avoid partnering with OEMs that care nothing about Mozilla supposed commitment to "openness".

Comment 46

[Felipe D.]

Yes, "we", but you do realize the ZTE Open launched in Colombia/Venezuela/Spain as a consumer phone a couple of weeks before "we" did through eBay?  Who is to say consumers there are not experiencing this issue? Are you expecting just anyone to figure out how to fix the issue themselves?

But what do I know. Maybe we are the only ones experiencing this issue. 

In any case, xiongping said they would be investigating this and letting us know. I am just asking him for the official answer. Is it (a) no, we cannot fix the broken OTA due to the corrupt build.prop, or (b) we will be fixing the broken OTA soon?

Comment 47

[Ben Kelly [:bkelly, not reviewing]]

I believe we recently got feedback that you can return your phone if it was damaged during an OTA update.  See the last paragraph on this page:

  https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Developer_phone_guide/ZTE_OPEN

Quote:

"If your phone was unfortunately damaged during the FOTA update because of ZTE providing false or wrong update files, you can send your phone back to the eBay store you purchased it from in exchange for a new one. You may contact the store owner directly via an eBay message."

I'm sorry for the inconvenience and that it has taken so long to investigate it!

Comment 48

[znmeb]

What's the latest on this? I have a ZTE Open that was updated on September 18 and claims to be 1.0.0B02. Is it? Should I root it and start installing Firefox OS manually?

Comment 49

[jezra]

(In reply to znmeb from comment #48)
> What's the latest on this? I have a ZTE Open that was updated on September
> 18 and claims to be 1.0.0B02. Is it? Should I root it and start installing
> Firefox OS manually?

You should follow the instructions at https://support.mozilla.org/en-US/questions/971252?page=2#answer-484863 which in theory should remove the lousy lock that ZTE implemented in the bootloader. After that, you should be able to compile and install a Firefox OS image by following instructions at https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS

Hopefully, testing images for newer version of Firefox OS will be available for download from Mozilla or ZTE. However, based on the amount of time it took to remove the unnecessary bootloader lock, it may be a long time before testing images are made available to us.

Comment 50

[znmeb]

(In reply to jezra from comment #49)
> (In reply to znmeb from comment #48)
> > What's the latest on this? I have a ZTE Open that was updated on September
> > 18 and claims to be 1.0.0B02. Is it? Should I root it and start installing
> > Firefox OS manually?
> 
> You should follow the instructions at
> https://support.mozilla.org/en-US/questions/971252?page=2#answer-484863
> which in theory should remove the lousy lock that ZTE implemented in the
> bootloader. After that, you should be able to compile and install a Firefox
> OS image by following instructions at
> https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/
> Building_and_installing_Firefox_OS
> 
> Hopefully, testing images for newer version of Firefox OS will be available
> for download from Mozilla or ZTE. However, based on the amount of time it
> took to remove the unnecessary bootloader lock, it may be a long time before
> testing images are made available to us.

OK ... I'll try that. I'm not really planning to do much with the ZTE Open at the lower levels at this point. I really want it as an app development dogfooding device. I'm in Portland, Oregon, and T-Mobile's coverage is spotty in the outlying areas anyhow. I have a Geeksphone Peak+ on order and will port the phone over to that when it shows up, then root the ZTE. ;-)

Comment 51

[BMO Automation]

Firefox OS is not being worked on