Server crash!!!

[mdre]

Our database crash with this error:

Caused by: com.orientechnologies.orient.core.exception.OStorageException: Page 2202 is broken in file interviniente_10.pcl
	DB name="heimdall"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.pageIsBroken(OAbstractPaginatedStorage.java:4109)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.callPageIsBrokenListeners(OWOWCache.java:569)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.assertPageIsBroken(OWOWCache.java:2409)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.loadFileContent(OWOWCache.java:2351)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.load(OWOWCache.java:1074)
	at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.lambda$doLoad$1(AsyncReadCache.java:226)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
	at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.doLoad(AsyncReadCache.java:220)
	at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.loadForRead(AsyncReadCache.java:126)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPageForRead(ODurableComponent.java:143)
	at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.internalReadRecord(OPaginatedClusterV2.java:663)
	at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.readRecord(OPaginatedClusterV2.java:635)
	at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.readRecord(OPaginatedClusterV2.java:620)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:5145)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:4857)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1919)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OEnterpriseLocalPaginatedStorage.readRecord(OEnterpriseLocalPaginatedStorage.java:1188)
	at com.orientechnologies.orient.core.db.document.SimpleRecordReader.readRecord(SimpleRecordReader.java:24)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeReadRecord(ODatabaseDocumentEmbedded.java:1345)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:213)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:80)
	at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeReadRecord(OConnectionBinaryExecutor.java:306)
	at com.orientechnologies.orient.client.remote.message.OReadRecordRequest.execute(OReadRecordRequest.java:98)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:355)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239)
	at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)

Are there any way to fix it? . REPAIR DATABASE fail.

[tglman]

Hi,

This looks like one of the record is broken, I do not think you can recover the single record, but it should be possible to truncate the specific record and still read the other data from the database.

Is this the only error you have in your database ?

Regards

[tglman]

Hi,

what does it mean it crashes ? there are many possible errors situation that you can get when you get this error. for example:

• The server fail to start
• The server start but when I try to open a session it fails to open the session
• I tried to write some data and I get that error
• When I run a query I do get that error
• When I run a specific query I do get the error
• I just saw this stack trace in the log but I cannot trace the error anywhere else

Depends on the situation you can do different operations, from what I see from the error it does seems that there is a specific error when reading a specific record, I do not think you can "correct" that specific record, the best way to correct the error would that would be identify the id of the record and truncate it with truncate record #record:id , this will cause the loss of the specific record. This also assume that you can start the server and open database, that I cannot guess from your message.

It may be possible to recover the data of the specific record from a deeper inspection from someone with a deep knowledge in OrientDB, but probably is easier just to see if you have any of that in backups.

[mdre]

Yesterday the database eat the whole memory and stop to respond. After a wait, we decide to emit a "service stop" and a "service start", but it never start again.
The log show that error again and again and again.....
This is the log:
orient-server.log.0.txt

[mdre]

I could connect with the console and I tried to run a repair database --fix-graph but it didn't help.

[mdre]

any clue please??
I'm reading the sources but it is a very hard way to find a solution.

[tglman]

Hi,

So the repair database do not handle this cases, reading the logs the error happen when reading a record the best way to fix this is check your log on the client side and try to identify what record is broken and truncate it.

Also reading the log you can see that the cluster with the issue is interviniente_10 so if you can connect with the console you can run the query select from cluster:interviniente_10 it will probably return some results till it fails, you can try to then run some query like select from #record:id that are slight bigger than the one you see in the results, like if you see a record id like for example #30:20 you can then try to run the query with #30:21, #30:22, #30:22 ecc and see which one fail and than truncate it with truncate record #record:id

There is no automatic procedure in the command line to handle this case automatically

[mdre]

Perfect! I will try.

[mdre]

Well, finally I could get open the database removing the wal files, but when I try to rebuild the index, I get this error:

2025-11-05 08:33:28:088 INFO  {db=heimdall} - Rebuilding index heimdall.Investigacion.uuid (estimated 2.895.332 items)... [OIndexRebuildOutputListener]
2025-11-05 08:33:37:701 WARNI {db=heimdall} Error deserializing record with id #4581:425 send this data for debugging: AYIENm91dF9DYXJwZXRhRmlzY2FsX3ByaW5jaXBhb
BwWOm91dF9DYXJwZXRhRmlzY2FsX3Jlc3BvbnNhYmxlHBYyb3V0X0NhcnBldGFGaXNjYWxfbGVnYWpvcxwWT/Yr+QIm/QIOYwwKX19hY2yKAgxZSv8CAoMDQvsCDIUDBDBpbl9FcXVpcG9GaXNjYWxfY2FycGV0Y
XMcFjhvdXRfQ2FycGV0YUZpc2NhbF9kb2N1bWVudG9zHBY4b3V0X0NhcnBldGFGaXNjYWxfYXVkaWVuY2lhcxwWUm91dF9DYXJwZXRhRmlzY2FsX2RpZ2l0YWxpemFkb3NQZW5kaWVudGVzHBYAys/S9wPBo/b2A
QAAAADKz9L3A8Oj9vYBAAAAAMrP0vcDxaP29gEAAADyK3lhc21pbiByb2RyaWdvIGdyYWRvIGFybWFzIHByZWN1cnNvcmVzIHlhaXIgZW1pbGlhbm8gZGVsIDI3ODgxMzg3IDUwMDM4NTg0IHByb2R1Y2Npw7NuI
GpvbmF0YW4gMzgzNzM0OTkgZ3JhY2llbGEgYWxlamFuZHJhIDQ0MDY1MjQ2IGxleSAyMzczNyBnb256YWxvIHlvYW5hIHByaW1hIGdhYnJpZWwgZ29tZXogNDA2MTk5MTcgNDAzNTk2MTIgc2ViYXN0aWFuIC0gc
GVyc29uYXMgbGFuZHJpZWwgNDY1Mzg0ODIgaW51bmRhY2nDs24gYXYgZmluZXMgamF2aWVyIDM0NzMwNjM3IGxsYW5vcyBlc3R1cGVmYWNpZW50ZXMsIGFsZGFuYSBtYXJxdWV6IGx1aXMgNDQyOTI2NjYgbWFpZ
GFuYSAzNzcxNDg4NCB2YWxsZWpvcyBkaXN0cmlidWNpw7NuLCBtaWd1ZWwgZ3JpZ2VyYSBqb2FxdcOtbiAxMDAsIDQzMTI3NTExIGFyYWJpYW4gMTY0MTg3NzIgam9hcXVpbiBwb3IgZGFybmlyIDQ3NTA5NTYyI
GNpdmlsIG1hYmVsIGNvbWVyY2lhbGl6YWNpw7NuIHRob21wc29uIGNvbWVyY2lhbGl6YWNpb24gZml0YSAzNzM1OTc3MiA0ODI1Mjc2OSBmZXJuYW5kbyBjdWFscXVpZXIgc3UgYWxtYWNlbmFtaWVudG8gMzk1M
DE4NjYgMTYwNjY0OTAgc2VndXJpZGFkIHNvbGVkYWQgam9zZSBqb2FuYSBub3JtYSBhbGVqYW5kcm8gbyBpdGF0aSBlcmlrYSBndXN0YXZvIHkgNDcwMzc4OTAgZmFiaWFuIGVzY29iYXIgNDMyMzY0OTAgY2Fyb
G9zIGRlIG1pbGFncm1pbGFncm9zIGh1Z28gYWxiYXJpw7FvIGRpIDQyOTI2NDQzIGVsaXNhYmV0IGp1YW4gZ2FzdG9uIGFsb25zbyAyODIwMzYyOCAzMTYxMzAxMSBtZXphIGJlbml0ZXogZmVybmFuZGV6IGNvb
WVyY2lhbGl6YWNpw7NuLCBhbmRyZWEgc2FuY2hleiBnaW1lbmV6IGdyw7xuaW5nIGVudHJlZ2Egam9lbCByZWdpbm8gNDQ5OTQ5NjAgcmV0YW1vem8gZnJhbmNvIGVuIHJvYmVydG8gYW5kcmVzIGF2YWxsZSAyO
DIgZXNwaW5kb2xhIDI4NiBleHRvcnNpw7NuIG9ydGlnb3phIGRlbGl0b3MgcGludG9zIDQ1NDkwNDkwIG5vZ3VlcmEgNDUyMTg0NjQgZXh0b3JzaW9uIDQ4NzM4NDA3IDM5ODE4MDM2IGluY2VuZGlvcyByb2phc
yA5NTUzMjgzNiBnaWdsaW9uZSBwYW9sYSBwZXJleiAzOTExOTIzNiB1dGlsaXphY2nDs24gYWJyaWwgZ29uemFsZXogZ2FsdmV6IDM4MTMyMzU2IGluY2VuZGlvIGRpZWdvIG1hcmlhIHRob21hcyB5b2xhbmRhI
HJpY2FyZG8gbcOhcyBvdHJhIGFsYmVydG8gcGFnbywgbG9wZXogYWx2YXJleiBtYXJjb3MgZGFtaWFuIDM4MTMyOTA2IGNyaXN0aWFuIHRhbmlhIGdpc2VsYSByb2RyaWd1ZXogY29tZXRlcmxvcyBsZW9uYXJkb
yBsdWRtaWxhIGFudG9pbmUgMjQ0ODg0OTIgZ2lhbmVsYSAzNjMzMzUyNiBleHBsb3Npw7NuIGhvcmFjaW8gZG9sb3NvcyBtaWNyb3RyYWZpY286IHF1w61taWNvcyBtaWx0b24gc3RlZmFubyBnbGFkeXMgMjk0M
TExNTEgcm9tw6FuIGZ1ZWdvIDQwMjQ2NzgzIDMxMzAwODc4IDQ3MzI5NTcxIDQwODQzNjI5IDEwOTg2MzY0IG9zY2FyaXogbWFyY2lhIDQ0NzY2MDM4IHJ1YmVuIG5pY29sYXMgcm9jaW8geWFtaWxlIG1pY3Jvd
HJhZmljbyB5YW1pbGEgdGVuZW5jaWEgb3JnYW5pemFkYXMgbm9lbWkgcHVlc3RvcyB2ZXJhIGxlb24gZXZlbHluIGlzYWlhcyBtYXJpbiBuYWlsYSBiw7pua2VyIGplc3VzIG1pcXVlYXMgc2FudGlhZ28gZXZlb
GluIHphcmF0ZSB2YW5pbmEgMzEzMzk0MDYgcm9jaGEgcm9tZXJvIDQwMDM4MDA3IDI2OTcwMTQ2IGFncmF2YWRvcyAzMDQ4ODU3MSAxODQ0MTQ1NyByb2RhIGZhYnJpY2FjacOzbiBhbHVzdHJpemEgZG9taWNpb
GlvIDQzNDI0MjY3IGphY2tlbGluZSA5NTgxNTAwMyAxODc3ODU1OCB4YXZpZXIgbm9ndWVzIHDDumJsaWNhIDMxOTgxNzkzIDI0OTAwNTY0IHJpdmVybyB3YWx0ZXIgYmVsZW4gcmVuem8gZ2FsZWFubyBvcmVsb
GFubyA0NTI2NTc0OSBnb2Jlcm5hZG9yIGNlbGVzdGUgMzc3MDA1MDAgMjk3MDMzODAgdGVudGF0aXZhIGxhZHJpZWwgZ3J1bm5pbmcgYWd1c3RpbiBtaXJjbyBlbGlvbmVsIDQyMiBsYSBpYXJhIDM3NjQ4MDg3I
GFndXN0w61uIDQ0MDY1MTc2IGFyb2NoYSBlcmljYSA0MDAzNjYwOCBjYWxpZmljYWRhIG5hdGFuYWVsIDQ0OTE4Mjg4IGNvbiBvdHJvcyBwYWJsbyBsaW9uZWwgbHVjYXMgc2ltcGxlIGFuZ2VsIGZsb3JlbmNpY
SBiZXJsYXJpIDIyOTA4ODU5IGVsaWFuYSAzODEzMjYxMiB1c28gZXVnZW5pbyAyNzI5MDg5MyAzNzI2NTA1NCBrYXJlbiB2YW5lc2Egbmljb2zDoXMgbW9yZW5hIG5haHVlbCBheWVsw6luIHZlbGF6cXVleiB0cmVzIDQ4OTcyNjg0IGVzdHJhZ29zIDI3NjkwNDQ2IDEyNTAwNzc2IHNvc2Egcm9zYXJpbyAzMTg4MDkzNSAzMTg4MDkzNiBjYXJvbGluYSBmbG9yZXMgZWRpc29uIGJlYXRyaXogMzY4NzkzNDAgaGVjdG9yIHRoaWFnbyBhcm1hIGx1Y2lhbmEgMjgzMzEzMTYgZ2FyY2lhIGRhbmllbCBsZW9uZWwgeWFuaW5hIHVyaWVsIGNvbWVyY2lvIDM3MjY1ODQ4IDM1MTgyNTc0IGlsZWfDrXRpbWEgeWlzc2VsaSAyMjgzMzQ4MSBwcmlzY2lsYSBzaWx2aW5hIGx1Y2lhbm8gY2FzdHJvIG1hcmNlbG8gYWdyYXZhZG8gdHJhbnNwb3J0ZSAzNDMzNzczNyAzNjU0NTA4MyAzOTUwMjM0MiBtdcOxb3ogY2VzcGVkZXMgYnJpdG8gbGF1cmEgbWFyaWFuZWxhIGV2YSBheWFsYSA0NjI5NTY3NSBwYXJhIGRhdmlkIGNvbnRyYSAzMDIzNDc0MSBpbnRlcnZlbmNpw7NuIHNhbiBzYW50aW5vIDM5MjQ3NzQ1IGZpam9zIGd1ZXJyYSBtYXJpZWxhIG1lZHJhbm8gZXN0dXBlZmFjaWVudGVzIDI4NTU3NjUxIHBlZHJhemEgZXVnZW5pYSBhbmFsaWEgMTAwIG1hdGVyaWEgY2FybWVuIHZpbGxhIHZlZ2EgbWF4aW1pbGlhbm8gMzQ5MzkxMzIkUk9EQS1MQU5EUklFTC1KQU5PDEdSVVBBTJqzhKCPZgYHSDFmZDljYzFiLTEzZjktNDNmYi04ZWM2LTVjMTU2YjYxZjNiYwH+////DwdIMWZjNWUzNjYtY2UzNC00ZDk5LWI5MDYtMWYzYTFkMjkzYTBkAf7///8PB0gzMjAzMGY5My0wMjBhLTRmZjMtOGNmOC00ZmUwOTc4NWQ4ZTQB/v///w9IZWE5YzM3MzUtYmUwOS00M2Q3LTk2MDgtZWUyZWUyNWRiZGE1BjRDYXN0cm8sIEFndXN0aW5hIChhY2FzdHJvKdTLoerAZpYEAMrP0vcDzaP29gEAAAAAys/S9wPb09L3AQAAAADKz9L3A4H9vfkBAAAAAMrP0vcDuf7s/QEAAAA=  [ORecordSerializerBinary]
2025-11-05 08:33:37:701 INFO  {db=heimdall} --> OK, indexed 245.310 items in 9.614 ms [OIndexRebuildOutputListener]
2025-11-05 08:33:37:702 SEVER {db=heimdall} Error during index rebuild [OIndexUnique]
java.lang.StringIndexOutOfBoundsException: offset 3279, count -1, length 3413
        at java.base/java.lang.String.checkBoundsOffCount(String.java:4591)
        at java.base/java.lang.String.<init>(String.java:523)
        at java.base/java.lang.String.<init>(String.java:487)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.HelperClasses.stringFromBytes(HelperClasses.java:171)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.HelperClasses.readString(HelperClasses.java:139)

and if I try to remove that node I get:

orientdb {db=heimdall}> truncate record #4581:425

Truncated 
Error: java.util.IllegalFormatConversionException: d != java.util.ArrayList

[tglman]

Hi,

Actually if you remove the WAL files, it can cause this sort of failures, so I would definitely suggest to not do that.

I would double check these errors to see if I can find some useful info there