[arthas] UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
问题描述
[arthas@1]$ trace cn.xx.dataservice.biz.dataservice.controller.v1.CommonSearchController datasetQuery
Affect(class count: 1 , method count: 1) cost in 262 ms, listenerId: 1
Enhance error! exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
error happens when enhancing class: class redefinition failed: attempted to change the schema (add/remove fields), check arthas log: /root/logs/arthas/arthas.log
# tail -100 /root/logs/arthas/arthas.log
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] ACTIVE
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046] REGISTERED
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046] BIND: /127.0.0.1:8563
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046, L:/127.0.0.1:8563] ACTIVE
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848] REGISTERED
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848] BIND: local:arthas-netty-localaddress
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848, L:local:arthas-netty-localaddress] ACTIVE
2022-11-24 13:57:21 [arthas-binding-thread] INFO c.t.a.core.server.ArthasBootstrap -as-server listening on network=127.0.0.1;telnet=3658;http=8563;timeout=6000;
2022-11-24 13:57:21 [arthas-binding-thread] INFO c.t.a.core.server.ArthasBootstrap -as-server started in 1285 ms
Arthas server already bind.
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] READ: [id: 0x082ba92d, L:/127.0.0.1:3658 - R:/127.0.0.1:39284]
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] READ COMPLETE
2022-11-24 13:57:22 [arthas-NettyHttpTelnetBootstrap-3-2] INFO c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
2022-11-24 13:57:35 [arthas-command-execute] INFO c.t.arthas.core.advisor.Enhancer -enhance matched classes: [class xxxx.controller.CommonSearchController]
2022-11-24 13:57:35 [arthas-command-execute] ERROR c.t.arthas.core.advisor.Enhancer -Enhancer error, matchingClasses: [class xxxx.controller.CommonSearchController]
java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:396)
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.enhance(EnhancerCommand.java:168)
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.process(EnhancerCommand.java:115)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2022-11-24 14:00:07 [arthas-NettyHttpTelnetBootstrap-3-2] ERROR io.termd.core.io_error -Reported io error => closing
java.lang.NoClassDefFoundError: one/profiler/AsyncProfiler
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.shouldOverridesName(AnnotatedCommandImpl.java:33)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.name(AnnotatedCommandImpl.java:51)
at com.taobao.arthas.core.shell.system.impl.InternalCommandManager.getCommand(InternalCommandManager.java:131)
at com.taobao.arthas.core.shell.system.impl.InternalCommandManager.getCommand(InternalCommandManager.java:41)
at com.taobao.arthas.core.shell.system.impl.JobControllerImpl.createProcess(JobControllerImpl.java:154)
at com.taobao.arthas.core.shell.system.impl.JobControllerImpl.createJob(JobControllerImpl.java:88)
at com.taobao.arthas.core.shell.system.impl.GlobalJobControllerImpl.createJob(GlobalJobControllerImpl.java:57)
at com.taobao.arthas.core.shell.impl.ShellImpl.createJob(ShellImpl.java:98)
at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.createJob(ShellLineHandler.java:83)
at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.handle(ShellLineHandler.java:61)
at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.handle(ShellLineHandler.java:17)
at com.taobao.arthas.core.shell.handlers.term.RequestHandler.accept(RequestHandler.java:22)
at com.taobao.arthas.core.shell.handlers.term.RequestHandler.accept(RequestHandler.java:10)
at io.termd.core.readline.Readline$Interaction.end(Readline.java:231)
at io.termd.core.readline.Readline$Interaction.access$000(Readline.java:187)
at io.termd.core.readline.Readline$2.apply(Readline.java:487)
at io.termd.core.readline.Readline$Interaction.handle(Readline.java:272)
at io.termd.core.readline.Readline$Interaction.access$200(Readline.java:187)
at io.termd.core.readline.Readline.deliver(Readline.java:121)
at io.termd.core.readline.Readline.access$500(Readline.java:38)
at io.termd.core.readline.Readline$Interaction$2.accept(Readline.java:435)
at io.termd.core.readline.Readline$Interaction$2.accept(Readline.java:429)
at io.termd.core.tty.TtyEventDecoder.accept(TtyEventDecoder.java:93)
at io.termd.core.tty.TtyEventDecoder.accept(TtyEventDecoder.java:26)
at io.termd.core.tty.ReadBuffer.accept(ReadBuffer.java:44)
at io.termd.core.tty.ReadBuffer.accept(ReadBuffer.java:28)
at io.termd.core.io.BinaryDecoder.write(BinaryDecoder.java:117)
at io.termd.core.io.BinaryDecoder.write(BinaryDecoder.java:65)
at io.termd.core.telnet.TelnetTtyConnection.onData(TelnetTtyConnection.java:139)
at io.termd.core.telnet.TelnetConnection.flushData(TelnetConnection.java:267)
at io.termd.core.telnet.TelnetConnection.flushDataIfNecessary(TelnetConnection.java:257)
at io.termd.core.telnet.TelnetConnection.receive(TelnetConnection.java:108)
at io.termd.core.telnet.netty.TelnetChannelHandler.channelRead(TelnetChannelHandler.java:47)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at com.alibaba.arthas.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at com.alibaba.arthas.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at com.alibaba.arthas.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: one.profiler.AsyncProfiler
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at com.taobao.arthas.agent.ArthasClassloader.loadClass(ArthasClassloader.java:34)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 52 common frames omitted
[arthas@1]$ sm -d xxxx.controller.CommonSearchController
declaring-class xxxx.controller.CommonSearchController
constructor-name <init>
modifier public
annotation
parameters
exceptions
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
constructor-name <init>
modifier private
annotation
parameters xxxx.controller.CommonSearchController$auxiliary$OrIwA4e9
exceptions
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
method-name datasetQuery$original$1pV75W4b
modifier private
annotation
parameters java.lang.String
java.lang.String
xxxx.common.dto.page.PageRequest
return xxxx.common.dto.CommonResponse
exceptions xxxx.common.exception.CommonException
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
method-name datasetQuery
modifier public
annotation org.springframework.web.bind.annotation.PostMapping,io.swagger.annotations.ApiOperation
parameters java.lang.String
java.lang.String
xxxx.common.dto.page.PageRequest
return xxxx.common.dto.CommonResponse
exceptions xxxx.common.exception.CommonException
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
method-name datasetQuery$original$1pV75W4b$accessor$yu3zRiZn
modifier final
annotation
parameters java.lang.String
java.lang.String
xxxx.common.dto.page.PageRequest
return xxxx.common.dto.CommonResponse
exceptions xxxx.common.exception.CommonException
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
method-name getSkyWalkingDynamicField
modifier public
annotation
parameters
return java.lang.Object
exceptions
classLoaderHash 66629f63
declaring-class xxxx.controller.CommonSearchController
method-name setSkyWalkingDynamicField
modifier public
annotation
parameters java.lang.Object
return void
exceptions
classLoaderHash 66629f63
Affect(row-cnt:7) cost in 33 ms.
问题原因
出现这个异常的原因是很明确的，在更新字节码时，增加/删除了field。可能有下面的原因：
- JDK的bug，尽量用最新的JDK版本，特别是JDK8
比如： redefine热更新报错 #969 (comment)
- 其它的java agent修改了字节码，增加了 field。
比如jacoco，这个代码测试覆盖率工具，它会在类里增加一个static field来记录执行信息。
- 其它的java agent修改了字节码，但是在retransform时，生成同样的字节码
比如skywalking： 无法trace 和watch加载skywalking7 的jar中的类方法 #1141
- 在执行arthas redefine/retransform命令时，用户自己想替换的字节码有修改
可能是用户本地的JDK版本和线上的JDK版本不一致导致的
可能是用户自己修改了源码
解决方法
暂时关闭 skywalking
X参考文献
- class redefinition failed: attempted to change the schema (add/remove fields) 分析排查 - github/arthas 【推荐】
- watch命令报错class redefinition failed: attempted to change the schema (add/remove fields) - github/arthas
- k8s中使用arthas：Enhance error! exception: java.lang.UnsupportedOperationException: class redefinition f... - CSDN
- 线上排查神器arthas总结 - CSDN
- 解析Arthas协助排查线上skywalking不可用问题_java - AB教程网
本文作者：
千千寰宇
本文链接： https://www.cnblogs.com/johnnyzen
关于博文：评论和私信会在第一时间回复，或直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
日常交流：大数据与软件开发-QQ交流群: 774386015 【入群二维码】参见左下角。您的支持、鼓励是博主技术写作的重要动力！
本文链接： https://www.cnblogs.com/johnnyzen
关于博文：评论和私信会在第一时间回复，或直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
日常交流：大数据与软件开发-QQ交流群: 774386015 【入群二维码】参见左下角。您的支持、鼓励是博主技术写作的重要动力！
浙公网安备 33010602011771号