"decrypt dat ass." - and a file to download named "smelf". That was the introduction to the current challenge. file reported a 64-bit elf binary.

The first execution without any arguments ends with a segmentation fault. A new try with "AAAA" for arguments prints "Wrong!". radare shows us the reason for this behavior. The bin needs an argument with a length of up to 29 characters.

push rbp
mov rbp, rsp
push rbx
sub rsp, 0x28
mov [rbp-0x28], rdi
mov [rbp-0x2c], esi ; (0x000000d4)
mov rax, [rbp-0x28]
mov rdi, rax
call 0x4004c0  ; 1 = imp.strlen
cmp rax, 0x1d
jbe 0x40063f  ; 2 = 0x0040063f

If the length of the argument is in this range, some calculations are done with every character of this string. After each calculation  the result is compared to a byte from a byte sequence. If all results are correct, "Done!" will be printed. So we hope that we can get the key if we can find the correct input-string.

We extract the byte sequence and the calculation and write a little script which brute forces a 29 character input string with a backtracking-algorithm.

#!/usr/bin/env python

import sys
target = '\x4c\x10\x49\x00\x24\x09\x49\x36\x09\x05\x1e\x26'
target += '\x25\x4b\x00\x74\x65\x41\x00\x1e\x2a\x4b\x00\x1e\x2a\x4b\x4c\x48'
strings = []
# extracted calculation
def calc(char1,char2):
        si = chr(char1)
        a = char2
        d = a
        d >>= 7
        d >>= 6
        a += d
        a &= 3
        d -=a
        a <<= 3
        c = a
        di=0x12345678
        di >>= c
        a=di
        a ^=ord(si)
        a &= 0xff       # get one byte   
        return a

def brute_force(c,t_index, char_list):
        global target
        global sourcen
        for c2 in range(33,128):
                c2 = chr(c2)
                char_list_copy = char_list[:]
                char_list_copy.append(c2)
                result = calc(c,ord(c2))
                if chr(result) == target[t_index]:
                        if len(target)-1 != t_index:
                                brute_force(ord(c2),t_index+1,char_list_copy)
                        else:
                                strings.append(''.join(char_list_copy))
                                print ''.join(char_list_copy)

for c in range(33,128):
        c = chr(c)
        brute_force(ord(c),0,[c])

This script shows us 71 possible strings. Every string starts with "xF146_1$_".

xF146_1$_1f4734f394f834f8340$
xF146_1$_1f4734f394f834f8340(
xF146_1$_1f4734f394f834f8340,
xF146_1$_1f4734f394f834f83400
xF146_1$_1f4734f394f834f83404
xF146_1$_1f4734f394f834f83408
xF146_1$_1f4734f394f834f8340<
xF146_1$_1f4734f394f834f8340@
xF146_1$_1f4734f394f834f8340D
xF146_1$_1f4734f394f834f8340H
xF146_1$_1f4734f394f834f8340L
xF146_1$_1f4734f394f834f8340P
xF146_1$_1f4734f394f834f8340T
xF146_1$_1f4734f394f834f8340X
xF146_1$_1f4734f394f834f8340\
xF146_1$_1f4734f394f834f8340`
xF146_1$_1f4734f394f834f8340d
xF146_1$_1f4734f394f834f8340h
xF146_1$_1f4734f394f834f8340l
xF146_1$_1f4734f394f834f8340p
xF146_1$_1f4734f394f834f8340t
xF146_1$_1f4734f394f834f8340x
xF146_1$_1f4734f394f834f8340|
xF146_1$_1f4734f394f834f834|"
xF146_1$_1f4734f394f834f834|&
xF146_1$_1f4734f394f834f834|*
xF146_1$_1f4734f394f834f834|.
xF146_1$_1f4734f394f834f834|2
xF146_1$_1f4734f394f834f834|6
xF146_1$_1f4734f394f834f834|:
xF146_1$_1f4734f394f834f834|>
xF146_1$_1f4734f394f834f834|B
xF146_1$_1f4734f394f834f834|F
xF146_1$_1f4734f394f834f834|J
xF146_1$_1f4734f394f834f834|N
xF146_1$_1f4734f394f834f834|R
xF146_1$_1f4734f394f834f834|V
xF146_1$_1f4734f394f834f834|Z
xF146_1$_1f4734f394f834f834|^
xF146_1$_1f4734f394f834f834|b
xF146_1$_1f4734f394f834f834|f
xF146_1$_1f4734f394f834f834|j
xF146_1$_1f4734f394f834f834|n
xF146_1$_1f4734f394f834f834|r
xF146_1$_1f4734f394f834f834|v
xF146_1$_1f4734f394f834f834|z
xF146_1$_1f4734f394f834f834|~
xF146_1$_1f4734f394f834f83xZ#
xF146_1$_1f4734f394f834f83xZ'
xF146_1$_1f4734f394f834f83xZ+
xF146_1$_1f4734f394f834f83xZ/
xF146_1$_1f4734f394f834f83xZ3
xF146_1$_1f4734f394f834f83xZ7
xF146_1$_1f4734f394f834f83xZ;
xF146_1$_1f4734f394f834f83xZ?
xF146_1$_1f4734f394f834f83xZC
xF146_1$_1f4734f394f834f83xZG
xF146_1$_1f4734f394f834f83xZK
xF146_1$_1f4734f394f834f83xZO
xF146_1$_1f4734f394f834f83xZS
xF146_1$_1f4734f394f834f83xZW
xF146_1$_1f4734f394f834f83xZ[
xF146_1$_1f4734f394f834f83xZ_
xF146_1$_1f4734f394f834f83xZc
xF146_1$_1f4734f394f834f83xZg
xF146_1$_1f4734f394f834f83xZk
xF146_1$_1f4734f394f834f83xZo
xF146_1$_1f4734f394f834f83xZs
xF146_1$_1f4734f394f834f83xZw
xF146_1$_1f4734f394f834f83xZ{
xF146_1$_1f4734f394f834f83xZ

This start-sequence looks like leet speak for "Flag_is_". So  we have 71 possible flags. We interpret the starting 'x' as a delimiter and look for a key, which ends with an 'x'. So we have only one string, which should be the correct flag.

xF146_1$_1f4734f394f834f8340x

We have to cut off "xF146_1$_" to get the right flag.

1f4734f394f834f8340