memops are nodes that have memory inputs, you can generically query them
for their memory input. We can also get rid of get_fragile_op_mem in
favor of get_memop_mem.