i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original
-
@sabik @xgranade @whitequark @porglezomp but they also changed the boundaries! "Input" checks all values from 2 to i+2 inclusive; but "ground truth" just trows i+2 iteration out.
@IngaLovinde @xgranade @whitequark @porglezomp
`i` starts from 1 in the "ground truth" version -
@IngaLovinde @xgranade @whitequark @porglezomp
`i` starts from 1 in the "ground truth" version@sabik @xgranade @whitequark @porglezomp ah I see, so the new i is just the old one + 1
-
@whitequark well the paper speaks of *code style* which is more than just formatting but also, shouldn't we welcome negative results in science?
@mc @whitequark do they actually even recognise it as a negative result though?
They seem to be presenting it as a positive one (looking at the abstract and conclusion) — but I admit I'm not familiar with the norms for writing this sort of paper.
-
i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original
the "ideal" (their choice of words) case is 64.2%
@whitequark amazing
-
i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original
the "ideal" (their choice of words) case is 64.2%
@whitequark Looking at https://upload.whitequark.org/1774306843-Duetcs_Code_Style_Transfer_through_Generation_and_Retrieval.pdf, Fig. 6:
Look at `bool ok, count = false;`: This leaves “ok” at an undefined value.
In any case that should print “YES”, the `ok = false;` line is never called, it’s undefined whether it prints “YES” or ”NO” (might even be different for each invocation).Neither the input nor the ground truth had that bug.
It looks like the researches did not notice it and considered it correct.
(64.2% …)It was obvious to me, would you have caught it?
-
@SRAZKVT @lu_leipzig there is a more real problem of "some people bounce off contributing if you ask them to fix style"
@whitequark @lu_leipzig yea, such as: the code being shit
-
@whitequark @lu_leipzig most tooling devs today seem to believe in a one size fits all with no configurability, kind of sad
also i think the problem of "but if every codebase isn't formatted exactly the same" is way overblown, once you start reading the code it really doesn't take long to adapt to a new style, barely a few minutes from my experience
@SRAZKVT @whitequark @lu_leipzig in general, I agree, but I almost wish I could have just told the software teams that I worked with a couple years ago “this is style for this language, just drank with it” instead of having hours of meetings about clang-format settings.
-
@SRAZKVT @whitequark @lu_leipzig in general, I agree, but I almost wish I could have just told the software teams that I worked with a couple years ago “this is style for this language, just drank with it” instead of having hours of meetings about clang-format settings.
@c0dec0dec0de @SRAZKVT @lu_leipzig I think it's different for corporate. I don't really care about most corporate code I touch (that isn't already OSS I maintain that is), it's completely whatever. I care a lot about this in projects I'm invested in success of
-
@whitequark So let me get this straight, IEEE thinks you should count it as a win if rewriting your code by vibing it has less than 15% better odds than a literal coinflip of reproducibility?
edited for clarity and to fix a typo
@disorderlyf @whitequark i think "ideal" here means "the best case scenario that we encountered under ideal conditions", as opposed to a target for how it should be
-
@c0dec0dec0de @SRAZKVT @lu_leipzig I think it's different for corporate. I don't really care about most corporate code I touch (that isn't already OSS I maintain that is), it's completely whatever. I care a lot about this in projects I'm invested in success of
@whitequark @SRAZKVT @lu_leipzig I get that. At the end of it, I was just like pick something, I don’t care. This will make your code more readable regardless what you pick and minimize diffs in some cases.
-
i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original
the "ideal" (their choice of words) case is 64.2%
@whitequark Uh, that's crazy O.o
-
@disorderlyf @whitequark IEEE and ACM don't do the research nor they think you to do things, they are publishers that own journals and conferences where researchers publish their work
@urixturing @whitequark I initially thought IEEE was like a standards body specifically for networking, like a hardware W3C. Regardless of who did the research, I thought this was their conclusion. It sounds like I was wrong on both parts
-
@disorderlyf @whitequark i think "ideal" here means "the best case scenario that we encountered under ideal conditions", as opposed to a target for how it should be
@sammy @whitequark I hope you're right
-
@urixturing @whitequark I initially thought IEEE was like a standards body specifically for networking, like a hardware W3C. Regardless of who did the research, I thought this was their conclusion. It sounds like I was wrong on both parts
@disorderlyf @whitequark that would be the IETF, who publishes the RFCs (networking standards like email or HTTP)
-
@disorderlyf @whitequark that would be the IETF, who publishes the RFCs (networking standards like email or HTTP)
@disorderlyf @whitequark but honestly I understand why it's very confusing
-
i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original
the "ideal" (their choice of words) case is 64.2%
@whitequark They also solved the halting problem? -
R relay@relay.infosec.exchange shared this topic
