Robert Miles Reporter
	Description • 5 years ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

I updated my previous version (about a month old) version of Firefox Quantum to Firefox Quantum 62.0.3 (64-bit).


Actual results:

Most or all of my VERY LARGE collection of sites (estimated at 40,000 sites; several levels of folders, with total number of folders almost 1,000) was lost.  At least two levels of folders remained); however, the few remaining sites appeared to be added by the Firefox update, and not from my previous collection of sites


Expected results:

All of the previous sites, and all of the previous folders, should be visible in the new version.  Sites were NOT previously saved with Firefox Sync - I was repeatedly unable to log in to ask about whether Firefox Sync has limits on the number of folders and the number of sites.

	Matthias Versen [:Matti]
	Comment 1 • 5 years ago

Did you already follow https://support.mozilla.org/en-US/kb/recover-lost-or-missing-bookmarks ?

	Mark Banner (:standard8)
	Comment 2 • 5 years ago

(In reply to Matthias Versen [:Matti] from comment #1)
> Did you already follow
> https://support.mozilla.org/en-US/kb/recover-lost-or-missing-bookmarks ?

Robert, did this help?

	Robert Miles Reporter
	Comment 3 • 5 years ago

It gave me some things to try, but none of them helped.

The "My Google Bookmarks are missing" section contains a bad link, so I can't tell if that link should have gone to something that would help.

There is only one profile, which appears to still contain all of the folders but very few of these folders still contain anything other than lower level folders.  I have deleted some of these now empty folders.

Other information I have found since reporting the problem:

I had previously been using a different synchronizer program to maintain a second copy of the bookmarks on my other desktop, until the company providing it shut down their server and dropped all support for that program many months ago.

Therefore, there was a second copy of most of the bookmarks - at least those stored before they dropped support.  At least some of those entered since then on the first desktop were lost, though. Installing the same update on the second desktop does not appear to have lost any of the bookmarks on that desktop.

	Mark Banner (:standard8)
	Comment 4 • 5 years ago

Robert, have you been able to restore from an older backup?

If you need to know how, please follow the "Restoring from backups" section here: https://support.mozilla.org/kb/restore-bookmarks-from-backup-or-move-them#w_restoring-from-backups

Note: there will likely be several in the list - you might need to look for the largest one rather than the latest.

	Robert Miles Reporter
	Comment 5 • 5 years ago

You're assuming, incorrectly, that there is a backup.

	Mark Banner (:standard8)
	Comment 6 • 5 years ago

Firefox should be saving backups approximately every day when idle. It will keep a maximum of 15 files. If there aren't any being saved, then either you've set "browser.bookmarks.max_backups" to 0, or there is a corruption problem in the database which has also stopped backups happening.

We have in the past seen corruption when XMarks has been used, I'm thinking this could be another case of it.

Do you have any backups of your computer (and hence Firefox profile) from before you upgrade to 62? If so, we may be able to analyse the database to see what went wrong.

In the meantime, please can you also try running the places verification: 

* Go to Help -> Troubleshooting Information
* Scroll down the page to find the "Places Database" section (it is near the bottom).
* Click the "Verify Integrity" button, this may take a few seconds to run.

Please copy & paste the output here.

	Robert Miles Reporter
	Comment 7 • 5 years ago

I found no sign that I have set browser.bookmarks.max_backups, so it should still be at the default value.

Where does Firefox put these backups?  Are the old ones still saved if the caches have been invalidated, even if no new one can be written?

My previous synchronizer was Xmarks.

Some very old backups, a few years old, from before my backup device stopped working.  At least the most recent year of those backups are inaccessible, but still before the last successful Xmarks synchronization.

> Task: checkIntegrity
+ The places.sqlite database is sane
+ The favicons.sqlite database is sane
> Task: invalidateCaches
+ The caches have been invalidated
> Task: checkCoherence
+ The database is coherent
> Task: expire
- The task queue was cleared by an error in another task.
> Task: originFrecencyStats
- The task queue was cleared by an error in another task.
> Task: vacuum
- The task queue was cleared by an error in another task.
> Task: stats
- The task queue was cleared by an error in another task.
> Task: _refreshUI
- The task queue was cleared by an error in another task.

	Marco Bonardo [:mak]
	Comment 8 • 5 years ago

XMarks is sadly known to have caused various corruptions of the database in the past, and that's likely what happened here. Due to that corruption Firefox is unable to create backups and likely the update failed. If you wish you can send your places.sqlite file to Mark or me through private mail and Firefox Send and we may be able to suggest some query to fix it.

Backups are usually inside a bookmarkbackups folder in the profile folder (easily reachable from the about:support page).

	Marco Bonardo [:mak]
	Comment 9 • 5 years ago

And actually, the file you should send is the places.sqlite.corrupt file that is likely present in your profile folder, since the new places.sqlite doesn't contain your bookmarks.

	Robert Miles Reporter
	Comment 10 • 5 years ago

No places.sqlite.corrupt file found.

I did find:

places.sqlite       46080 KB
places.sqlite-shm   32 KB
places.sqlite-wal   5541 KB

Are any of these useful?

I found the bookmarkbackups folder, with 15 backups in it.  Those from March and April are much larger than the more recent ones, but still smaller than places.sqlite.

I have upgraded to Firefox 63.0.

From a previous message about restoring bookmarks:

Click the 57 library icon , then click Bookmarks. Scroll to the bottom and click Show all Bookmarks.
In the Library window, click the Import and Backup button and then select Restore.

The first two sentences worked.  There was no Import and Backup button for the third sentence.

	Robert Miles Reporter
	Comment 11 • 5 years ago

For the third sentence, Import and Backup appeared on the second line of the header, but not as a button.  Clicking on the triangle just to the right of it, gave me a list of 15 backups I could restore.

I'm not ready to choose which to restore, since I'm most interested in the sites saved since the last successful Xmarks sync, not the many more sites which Xmarks synced onto my other desktop.

	Marco Bonardo [:mak]
	Comment 12 • 5 years ago

Store those backup files apart for now, so they won't be removed. Then you can always try restoring them and see. Note the restore process may take a while, and it doesn't have a progress bar (there's a bug filed about that), so I'd suggest to let it alone for a while...

	Robert Miles Reporter
	Comment 13 • 5 years ago

Verify Integrity:

> Integrity check
+ The database is sane
> Coherence check
+ The database is coherent
> Orphans expiration
+ Database cleaned up
> Vacuum
Initial database size is 1696 KiB
+ The database has been vacuumed
Final database size is 1696 KiB
> Statistics
Database size is 1696 KiB
user_version is 36
page_size is 32768
cache_size is -2048
journal_mode is wal
synchronous is 1
History can store a maximum of 66607 unique pages
Table moz_places has 1576 records
Table moz_historyvisits has 1945 records
Table moz_inputhistory has 0 records
Table moz_bookmarks has 6 records
Table moz_keywords has 0 records
Table sqlite_sequence has 1 records
Table moz_favicons has 5 records
Table moz_annos has 0 records
Table moz_anno_attributes has 1 records
Table moz_items_annos has 1 records
Table sqlite_stat1 has 14 records
Table moz_hosts has 238 records
Table moz_bookmarks_deleted has 0 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index sqlite_autoindex_moz_hosts_1
Index sqlite_autoindex_moz_bookmarks_deleted_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_hashindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Index moz_keywords_placepostdata_uniqueindex

	Mark Banner (:standard8)
	Comment 14 • 5 years ago

Hi Robert, I'm sorry for the delay in responding. Which version of Firefox did you generate the output in comment 13? That seems like either a strange old version, or the database version wasn't updated correctly.

It might be better to start off from scratch - remove places.sqlite from the profile, and then use whichever most complete backup you have (make sure you let import finish), and see if you get any further issues from there.

	Robert Miles Reporter
	Comment 15 • 5 years ago

Information lost on what version was used for comment 13.  I have now upgraded to Firefox Quantum 63.0.3 (64 bit), since the info in comment was generated.

	Mark Banner (:standard8)
	Comment 16 • 5 years ago

So is it all working now?

I'm not sure there's much we can do for this, seeing as we don't really know what caused it or why it occurred.

	Robert Miles Reporter
	Comment 17 • 5 years ago

I would say that the problem is not getting worse, rather than saying it is all working now.

Of the 15 backups found above, those for 2/21/2018 and 3/2/2018 are about the same size as the backups on my other computer (about 2.1 MB), so I assume that Xmarks synchronized these properly.  The one for 4/17/2018 is about 1.4 MB, and all those since are either 32 KB or 34 KB.

Xmarks support was dropped 5/1/2018, so the problem occurred before they dropped support and I uninstalled the Xmarks program.

Note - the dates of backups on the computer with the problem suggests that a backup will be made automatically only on days when I have made some change to the URLs list since the previous backup.

	Mark Banner (:standard8)
	Comment 18 • 5 years ago

Are you getting new backups made, and are they looking about the right sort of size?

	Robert Miles Reporter
	Comment 19 • 5 years ago

Computer with the problem currently will not boot, so I'm currently unable to check.

	Mark Banner (:standard8)
	Comment 20 • 5 years ago

Ok, thank you for the response. For now, I'll close this as incomplete. If there's more information on an actual problem, we can always re-open or handle it in a new bug.