News for hackers. Stuff that matters.

A comment on UniFi Network Application 6.5.54. Parent comment

Re:UniFi Network Application 6.5.54 by ComputerGuru on Sunday December 12, 2021 @10:32PM (#29536315)
Tell me about it. Any time the Debian machine I have running as a Unifi appliance loses power (on an XFS filesystem on a disk with write caching disabled), the mongodb database is absolutely hosed (despite no xfs corruption at all) and I need to wipe the data directory and restore from backup because mongo 3 is just such garbage. (I mean, mongodb is still garbage but it's improved a tiny bit in some reliability metrics.)
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by jd_mongodb on Monday December 13, 2021 @03:19AM (#29537764)
I would be curious to learn more about your setup. MongoDB should be able to recover in these situations...
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by ComputerGuru on Monday December 13, 2021 @08:41AM (#29539477)
Standard from packages. After the power failure the database is unusable and I get mongo startup errors ending with “newer versions of mongo may be able to fix errors like this automatically” and it dies.
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by jd_mongodb on Monday December 13, 2021 @12:01PM (#29541463)
Which package? Are you running a replica set or a single node? What specific version? Also which storage engine?

3.0 is a pretty old version. Most 3.x versions are end of life or unsupported.

[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by ComputerGuru on Monday December 13, 2021 @12:21PM (#29541750)
Latest 3.6, single node, WiredTiger storage engine. The fact that it’s severely outdated is exactly the point we are making.

I don’t know if it’s possible to override the version of a Debian package’s specified dependency - although I also don’t know if mongodb 5 is drop-in backwards compatible with a client written and released against v3.6.

[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by jd_mongodb on Monday December 13, 2021 @01:23PM (#29542564)
Can you share the command line and/or the config you use to start up. We don't support any single node use of MongoDB in production BTW. It can't recover as you have found out...
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by ComputerGuru on Tuesday December 14, 2021 @11:52PM (#29562252)
To clarify again, I'm not the one starting it - Unifi ships its network appliance as a debian package that depends on mongod and starts it up itself (and it's absolutely a single node configuration). Maybe you guys should have a talk with them :)

This is what monogd.log contains when started:

    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] MongoDB starting : pid=7531 port=27117 dbpath=/usr/lib/unifi/data/db 64-bit host=unifi
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] db version v3.6.20
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] git version: 39c200878284912f19553901a6fea4b31531a899
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] allocator: tcmalloc
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] modules: none
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] build environment:
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten]     distmod: debian92
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten]     distarch: x86_64
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten]     target_arch: x86_64
    2020-09-24T21:18:54.231-0500 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/lib/unifi/run" } }, processManagement: { pidFilePath: "/usr/lib/unifi/run/mongod.pid" }, storage: { dbPath: "/usr/lib/unifi/data/db" }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/usr/lib/unifi/logs/mongod.log" } }
    2020-09-24T21:18:54.231-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by jd_mongodb on Wednesday December 15, 2021 @07:23AM (#29564984)
It's using the WiredTiger storage engine which has good crash recovery support. How does the crash present?
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by ComputerGuru on Thursday December 16, 2021 @09:27PM (#29586974) 

    021-12-07T19:05:23.822-0600 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/lib/unifi/run" } }, processManagement: { pidFilePath: "/usr/lib/unifi/run/mongod.pid" }, storage: { dbPath: "/usr/lib/unifi/data/db", wiredTiger: { engineConfig: { configString: "cache_size=256M" } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/usr/lib/unifi/logs/mongod.log" } }
    2021-12-07T19:05:23.822-0600 I -        [initandlisten] Detected data files in /usr/lib/unifi/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
    2021-12-07T19:05:23.822-0600 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),cache_size=256M
    2021-12-07T19:05:24.339-0600 E STORAGE  [initandlisten] WiredTiger error (-31802) [1638925524:339440][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1638925524:339440][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
    2021-12-07T19:05:24.339-0600 E STORAGE  [initandlisten] WiredTiger error (0) [1638925524:339475][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1638925524:339475][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
    2021-12-07T19:05:24.339-0600 E STORAGE  [initandlisten] WiredTiger error (0) [1638925524:339483][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1638925524:339483][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
    2021-12-07T19:05:24.339-0600 E STORAGE  [initandlisten] WiredTiger error (0) [1638925524:339489][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1638925524:339489][1327:0x7f952058b400], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
    2021-12-07T19:05:24.340-0600 F STORAGE  [initandlisten] WiredTiger metadata corruption detected
    2021-12-07T19:05:24.340-0600 F STORAGE  [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
    2021-12-07T19:05:24.340-0600 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
    2021-12-07T19:05:24.340-0600 F -        [initandlisten] \n\n***aborting after fassert() failure\n\n
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by jd_mongodb on Friday December 17, 2021 @03:50AM (#29589581)
If I can get the unifi device and version I will follow up with unifi directly.
[ Parent | on HN ]
Re:UniFi Network Application 6.5.54 by ComputerGuru on Sunday December 19, 2021 @07:15PM (#29620101)
It's device agnostic; this is from the controller software that manages all the Unifi devices on your network. This was the latest version of the controller, available on their website as "UniFi Network Application 6.5.55 for Debian/Ubuntu Linux and UniFi Cloud Key" (Cloud Key is the appliance they ship that simply runs Debian w/ the package installed).

My email is mqudsi at neosmart dot net; I'd love if you could let me know what happens.

[ Parent | on HN ]